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ABSTRACT 

This paper attempts to lay a general framework for 
analyzing credibility gaps in evaluative information at various 
levels of organizational structure. The CIPP model is chosen as a 
point of reference in order to identify the type of evaluative 
information to which the concern of this paper is addressed. 
Credibility gaps are classified as personnel related and organization 
related. The latter, an Organizational Credibility Gap, exists at 
each organizational level where the same person is both accountable 
for a program's success or failure and responsible for the formal 
evaluation of that program. Three characteristics of such gaps are; 

(1) They are basically a "conflict of interests" relationship between 
the organizational roles; (2) They are separate from personalities as 
such; and (3) The essence of these gaps is that of potential 
incredulity. Four propositions which may serve as guidelines in the 
institutionalization of evaluation functions in educational 
institutions are; (1) Gaps can be delineated in a project's planning 
stages; (2) Gaps should be minimized; (3) The bridging of each gap 
should be planned; and (4) The strategy of internal evaluation with 
external audit provides the balanced use of resources for a 
long-range development and institutionalization of evaluation 
functions in educational institutions. A combination of the 
propositions supports an overall strategy that would support the 
long-range buildup of in house evaluation capabilities of educational 
institutions. (CK) 
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Accountability seems to be the key concept that focuses the issue 
implied by the title of this paper. The problems of the credibility of 
evaluative information passed from a funded project to this funding 
agency is a critical issue as far as funding agencies are concerned, but 
it is also a critical issue for the funded agency. Much of this paper 
is an effort to lay a general framework for analyzing credibility gaps 
in evaluative information at various levels of organizational structure. 
It is hoped that the general framework will provide a better way of look- 
ing at the more specific kind of credibility gap implied by the title of 
the paper. 

The CIPP model is chosen as a point of reference in order to identify 
the type of evaluative information to which the concerns of this paper 
are addressed (Stuff lebeafli, et.al. , 1971). Within the framework 
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of the CIPP model, product evaluation is the type of evaluative 
information of primary concern in this paper, although the cred- 
ibility gap between public schools and funding agencies, such as in 
reported results of needs assessment, availability of resources, 
implementation of planned strategies which would be specific examples 
of other types of evaluative information in the CIPP model, may also 
justifibly fall under the credibility gap concerns delineated in this 
paper. 

If credibility gaps may be classified by the sources which engender 
them, one major category would be personnel related. Such credibility 
gaps would be engendered by doubts as to the technical competence or 
personal honesty of the personnel providing the evaluative information. 

This category has to do directly with the perceived nature of the persons 

who are functioning in organizational roles. 

Another major category of credibility gaps would be those which are 

organization related. This category has to do with the organizational 
roles in which persons function rather than to the persons themselves. 

Since this type of credibility gap is germane to the manner of institu- 
tionalization of the evaluation functions, it may be termed an organiza- 
tional credibility gap. The simple criterion for determining the existence 
of such a gap may be stated as follows: An organizational credibility 

gap exists at each and every organizational level where the same person is 
both accountable for the success or failure of a program or project and 
responsible for the formal evaluation of that program or project. 
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Organizational credibility gaps are basically a "conflict of 
interests" relationship between the organizational roles. Several 
important characteristics of organizational credibilty gaps as thus 
defined should be kept in mind. First, there may be a multiplicity 
of such gaps in the "chain-of-command" of any given program or project. 

For instance, if the responsibility for evaluation of a project is 
merged with the chain-of-command responsibilty at the project director 
level, every level of the chain— of— command up to the top level of the 
organizational structure yields an organizational credibility gap. 

A second characteristic of organizational credibility gaps is that 
their existence is completely separate from personalities as such. An 
organizational credibility gap is a characteristic of the organizational 
structure and exists when the criterion applies, regardless of the persons 
performing in organizational roles. Therefore, when an organizational 
credibility gap is delineated no implication of dishonesty or other person- 
al characteristic is meant in relation to any personality acutally perform- 
ing in organizational roles. 

A third characteristic of organizational credibility gaps is that their 
essence is that of potential incredulity rather than actual incredulity. 

With respect to a given report of evaluative information, formal or informal, 
incredulity may not be in fact attached to the report by some administrative 
level in spite of the existence of organizational credibility gaps. An 
occasion for doubt exists, however, in the inherent "conflict of interests" 
situation of the organizational roles. 
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Figure 1 provides a comparative illustration of placement of project 
evaluation functions within an educational institution. In both examples 
(Types A and B) the immediate management of the project is five adminis- 
trative echelons removed from the governing board, and the "chain-of- 
command" structure is identical with respect to project operations. There 
is a substantial contrast between the two types, however, in the number of 
organizational credibility gaps created. This substantial difference is 
solely the function of organizational placement of the evaluation functions. 
In Type A, the project evaluation' team is assigned to, and under the direct 
control of the project director. According to the criterion of an organi- 
zational credibility gap described above, an organizational credibility 
gap is thus created at the level of the project director. Since every 
administrative echelon above the project director in the chain-of-command 
is also responsible for the success of the project as well as for control 
of the project evaluation team, the criterian shows that four other orga- 
nizational credibility gaps are also created by the merging of project 
control and evaluation at this level. 

In the Type B method of institutionalizing the project evaluation 
functions, the project evaluation team is assigned to provide evaluative 
information on the same project, indicated by the dotted line, but is under 
the administrative control of an administrator of evaluation services not 
in the project chain-of-command. This administrator reports directly to, 

and is directly under the administrative control of the chief administrator 



O 

ERIC 



4 



5 



Type A 



Type B 




"""Organizational credibility gap regarding evaluative information. 



Figure 1. — Illustration of organizational credibility gaps created by 
two alternate forms of institutionalizing the evaluation functions. Titles 
designate roles but may not necessarily be the aotual title of administrative 
personnel filling the roles. 
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(superintendent, agency head, college president, etc.) of the educa- 
tional position where the criterion for an organizational credibility 
gap applies. Therefore, an organizational credibility gap exists only 
between the chief administrator and the governing board. 

From the perspective of the chief administrator in Type B no orga- 
nizational credibility gaps are built into the flow of project evaluative 
information that comes to his desk; whereas in Type A he must recognize 
that the evaluative information has had to flow through four organizational 
credibility gaps before it reaches his deak. This does not necessarily of 
itself mean that the evaluative information is inaccurate, censored, or 
otherwise slanted to make the project look better than it is, but the 
risks of such being the case are greater. 

Figure 2 is an abbreviated organizational chart in a large city school 
district with reference to some of its operating programs and projects 
for which extensive evaluation was designed and implemented.^ It provides 
a convenient example of several key issues and problems related to 
organizational credibility gaps. 



•^•The organizational chart is based on operations in the Dallas Independ- 
ent School District in 1969 through 1971. The structure of different programs 
and projects within the organization was in considerable flux through this 
period of time. The chart presented in Figure 2, therefore, is an attempt 
to illustrate some of the various organizational relationships that existed 
during the period but is not necessarily completely accurate with respect 
to any one point of time in that period. During this period the author 
served as the Assistant Superintendent for Planning, Research, and Evaluation. 
The Dallas Independent School District encompasses most of the area of the 
eighth largest city in the nation by population (1970 census) and stands in 
about the midrange of the twenty or thirty largest school districts in the 
nation. 
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figure 2. — Example of various placement of project/program evaluation functions in a large city 
school district. 
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The organizational chart in Figure 2 illustrates various strategies 
for institutionalizing the evaluation functions. Until 1969, evaluation 
functions were primarily performed in the line of the chain-of-command, 
except for a system-wide achievement testing program which was centrally 
administered. In 1969, a new research and evaluation strategy was introd- 
uced into the system. The strategy was to upgrade "in-house" educational 
research and evaluation expertise in the system and to concentrate this 
expertise in a Department of Planning, Research, and Evaluation. 

Two primary objectives underlay the adoption of the new strategy: 

(1) to increase technical competence of the system to conduct education- 
al evaluation and (2) to increase separation between project/program 
operations and project/program evaluation. The strategy was not immed- 
iately, uniformly implemented, however. It was initially applied to 
specially funded projects, high interest projects and programs, and a 
few projects and programs where project and program managers voluntarily 
sought technical assistance from the newly formed department or volun- 
teered to relinquish evaluation functions to that department. In short, 

* it was a transition period and, thus, illustrates a comingling of the new 

and residual strategies. 

The placement of evaluation of the ESEA Title I program illustrated 
in Figure 2 created four organizational credibility gaps within the 
system: coordinator to director to associate superintendent to general 

superintendent to board of education. On the other hand, placement, of the 
evaluation functions for ESEA Title I programs under the Department of 
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Planning, Research, and Evaluation would have left only one organi- 
zational credibility gap within the system: general superintendent 

to board of education. 

When a new drug abuse education program was inaugurated, the 
responsibility for designing and implementing the evaluation of the 
program was assigned to the Department of Planning, Research, and 
Evaluation. Such an arrangement minimized the organizational cred- 
ibility gaps by leaving only one: general superintendent to board 

of education. Although the Assistant Superintendent of Planning, 

Research, and Evaluation did not report directly to the General 
Superintendent, but rather to the Associate Superintendent of Devel- 
opment, no organizational credibility gaps were created by this layer- 
ing of administrative echelons because the Associate Superintendent 
of Development was not in a chadjnt -of-command role in the drug abuse 
education program. 

The situation was different, however, for projects under the control 
of the Associate Superintendent of Development. As may be seen from 
the organizational chart in Figure 2, application o£ the criterion for 

t 

an organizational credibility gap would delineate an additional orga- 
nizational credibility gap at the level of the Associate Superintendent 
for Development for all such projects. For instance, the high school 
performance contract, which .involved two contractors external to the 
system, was administered by the Assistant Superintendent for Accountability. 
Responsibility for implementing the evaluation design was assigned to the 
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Department of Planning, Research, and Evaluation. From the perspective 
of the Associate Superintendent of Development, all organizational cred- 
ibility gaps regarding the evaluative information coming to his desk 
were eliminated by this arrangement. However, from the perspective of 
the General Superintendent, an organizational credibility gap existed 
with regard to scuh information, and from the perspective of the Board 
of Education two such gaps existed. 

Another projedt in the Division of Development involved additional 
complexities. This was the Dunbar Community Learning Center, which was 
a developmental site for development and/or piloting of various learning 
systems. On the upper side of the complexity continuum was an external 
private foundation providing substantial funds for the projects in the 
Center. On the lower end- of the continuum was the creation of a research 
and evaluation unit which was subject to chain-of-command control at the 
center level. This unit received technical assistance from the Depart- 
ment of Planning, Research, and Evaluation but was not under the admin- 
istrative supervision of that department. Thus, another organizational 
credibility gap was created, i.e., Center level to level of Associate 
Superintendent of Development. From the perspective of the Associate 
Superintendent of Development, one organizational credibility gap existed 
with respect to evaluation information supplied by the Center's research 
and evaluation unit: at the level of General Superintendent ,. two such 

gaps; at the level of the Board of Education three such gaps; and at the 
level of the external funding agency four such gaps* 
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So far, the consideration of organizational credibility gaps has been 
focused on gaps within a single institutional structure. Such credibility 
gaps are not solely a matter of internal concern. They are of concern not 
only to the chief administrator, but also to levels, agencies, funding 
sources, publics, and so forth, above and beyond the chief administrator. 
Those starategies which would tend to remove doubts about the quality of 
evaluative information at any levels) tend to increase its credibility 
at all levels. But what about credibility gaps beyond a single institu- 
tional structure, such as, state boards, agencies, and legislative bodies; 
regional and professional • accrediting associations; and external funding 
sources, both private and governmental? In principle, organizational cred- 
ibility gaps may exist in these relationships also. In addition, there 
may be multiple levels of organizational credibility gaps within these 
organizations. 

An example of an additional organizational credibility gap existing 
in relation to an external funding source was pointed out regarding the 

.1 

funding from a private foundation in Figure 2 . A similar point could be 
made in relation to the state and federal levels in relation to the ESEA 
Title I program. 

The same principle of keeping evaluation functions separated from 
project management responsibilities could be applied by external funding 
sources, private and governmental, as a condition for receiving funding. 
Such conditions have been imposed in some projects, such as the performance 
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contracts piloted by the Office of Economic Opportuntity beginning in 
1970. Some observers have even felt that this is a growing trend with 
the current push for accountability. It is the opinion of this author 
that such a trend would be detrimental to the overall substantial devel- 
ment and institutionalization of evaluation functions in educational 
institutions. The drive for accountability should not be allowed to over- 
whelm parallel functions of educational evaluation, especially if altern- 
atives are available for concurrently strengthening accountability as 
well as the other objectives of evaluation, such as continuous improve- 
ment of materials, technology, and personnel. Such a promising altern- 
ative .seems to be available in the two-stage model as an "internal" eval- 
uation and an "external" evaluation audit (see Stenner and Webster, 1971). 

The foregoing description and explication of organizational credibil- 
ity gaps provides the background for four general propositions which may 
serve as useful guidelines in the institutionalization of evaluation 
functions in educational institutions, as follows; 

1. Organizational credibility gaps can be delineated in the 
planning stages of a project or program. 

2. In the institutionalization of evaluation functions, organ- 
izational credibility gaps should be minimized insofar as 
other considerations permit. 

3. Every organizational credibility gap should be delineated in 
the planning stage of a project or program and the bridging 
of each gap should-be. planned. 
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4.. From the perspective of an external funding agency, the strategy 
of internal evaluation with external audit provides the best 
balanced use of resources for a long-range development and 
institutionalization of evaluation functions in educational 
institutions. 

Proposition One is based on the feasibility of applying the criterion 
for an organizational credibility gap to the proposed organizational 
structure of a project or program. 

Propositions Two and Three embody two basic proposals for dealing with 
delineated organizational credibility gaps in project/program planning 
stages. Proposition Two relates primarily to the placement of the evaluation 
functions in relation to accountability levels of a project/program. It 
simply means that consideration should be given to organization, or 
re-organization if need be, of reporting and control channels in such a 
manner that these channels will remain separated as far up the organi- 
zational levels as may be feasible in the light of other constraints, 
such as costs, efficiency, and effectiveness of providing evaluative inforr- 
mation. If it is judged that at some level these other constraints require 
the project/program evaluators to be ’’members of the team," it should be 
determined how broadly the "team" concept may be construed. For instance, 
in relation to Figure 2, evaluators assigned to provide evaluative information 
on the drug abuse education program were regarded as members of the team 
in the sense that they were still employees of the school district, yet they 
were still organizationally separate from the operational level of the 
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program. The balance point between being on the team and yet as orga- 
nizationally separate as feasible is difficult to judge and delicate to 
maintain. In fact, the judgment of the balance point may shift with the 
perspective at different organizational levels. 

The strategy of minimizing the existence of organizational credibility 
gaps minimizes excess effort in evaluation functions. If every organiza- 
tional credibility gap is to be bridged by some provision (Proposition 3), 
extra effort must be exerted — extra resources applied. The fewer gaps 
allowed to exist, the less the extra resources which must be applied. 

The alternative may at least theoretically exist to eradicate all 

organizational credibility gaps by attaching the function of providing 

evaluative information to the highest level of concern in the project. 

% 

For instance, in relation^ to Figure 2, all organizational credibility 
gaps could be removed from the perspective of the Board of Education 
if project/program evaluative information were institutionalized as an 
office or unit staffed by and directly responsible to the Board of 
Education or by contracting directly with outside firms or agencies to 
provide the evaluative, information directly to the Board of Education. 
This strategy could be expected to generate considerable resistance by 
all administrative and operational levels if it became a regular mode of 
operation. However, the need probably exists today, for at least some 
experimental applications of this model. The opportunity for slanting 
and censoring is great enough, even when qnlyv one credibility gap remains 
internal to the organizational structure, that some funding agencies 
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could experiment with this as an alternative model to contracting with 
a completely external evaluator which reports directly to the funding 
agency . 

Evaluative information is directed toward decision alternatives, but 
accountability concerns only represent one subset of the decision altern- 
atives, Another important subset relates to the efficient and timely use 
of evaluative information by various levels of decision makers— r-from 
operational level to top funding agency level. 

Thus, Proposition Pour is a proposal, perhaps even a plea, that comT> 
mittment of resources to evaluation be based on a strategy that would 
tend in the long run to build up the "in. house ,,; capabilities of educational 
institutions to conduct quality project/program evaluation as well as to 
serve the immediate concerns of accountablility at the funding source level. 
The proposition favors a strategy which would be similar to various ESEA 
guidelines wherein up to ten percent of the resources may be used for project 
evaluation and up to two percent for an independent evaluation audit. 

A combination of the propositions supports an overall strategy that 
would support the long-range build up of in house evaluation capabilities 
of • educational institutions, with a careful eye to minimizing internal 

organizational credibility gaps, and an independent evaluation audit 
that would bridge the "external” credibility gap or gaps between the funded 
institution and its funding sources. 
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